{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# RealtimeAgent with WebRTC connection\n",
    "\n",
    "\n",
    "AG2 supports **RealtimeAgent**, a powerful agent type that connects seamlessly to OpenAI's [Realtime API](https://openai.com/index/introducing-the-realtime-api). In this example we will start a local RealtimeAgent and register a mock get_weather function that the agent will be able to call.\n",
    "\n",
    "**Note**: This notebook cannot be run in Google Colab because it depends on local JavaScript files and HTML templates. To execute the notebook successfully, run it locally within the cloned project so that the `notebooks/agentchat_realtime_webrtc/static` and `notebooks/agentchat_realtime_webrtc/templates` folders are available in the correct relative paths.\n",
    "\n",
    "````{=mdx}\n",
    ":::info Requirements\n",
    "Install `ag2`:\n",
    "```bash\n",
    "git clone https://github.com/ag2ai/ag2.git\n",
    "cd ag2\n",
    "```\n",
    ":::\n",
    "````\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "## Install AG2 and dependencies\n",
    "\n",
    "To use the realtime agent we will connect it to a local websocket through the browser.\n",
    "\n",
    "To be able to run this notebook, you will need to install ag2, fastapi and uvicorn.\n",
    "````{=mdx}\n",
    ":::info Requirements\n",
    "Install `ag2`:\n",
    "```bash\n",
    "pip install \"ag2\", \"fastapi>=0.115.0,<1\", \"uvicorn>=0.30.6,<1\" \"flaml[automl]\"\n",
    "```\n",
    "For more information, please refer to the [installation guide](https://docs.ag2.ai/latest/docs/user-guide/basic-concepts/installing-ag2).\n",
    ":::\n",
    "````"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# !pip install \"ag2\" \"fastapi>=0.115.0,<1\" \"uvicorn>=0.30.6,<1\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Import the dependencies\n",
    "\n",
    "After installing the necessary requirements, we can import the necessary dependencies for the example"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "from logging import getLogger\n",
    "from pathlib import Path\n",
    "from typing import Annotated\n",
    "\n",
    "import uvicorn\n",
    "from fastapi import FastAPI, Request, WebSocket\n",
    "from fastapi.responses import HTMLResponse, JSONResponse\n",
    "from fastapi.staticfiles import StaticFiles\n",
    "from fastapi.templating import Jinja2Templates\n",
    "\n",
    "import autogen\n",
    "from autogen.agentchat.realtime.experimental import AudioObserver, RealtimeAgent"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Prepare your `llm_config` and `realtime_llm_config`\n",
    "\n",
    "The [`LLMConfig.from_json`](https://docs.ag2.ai/latest/docs/api-reference/autogen/llm_config/LLMConfig) method loads a list of configurations from an environment variable or a json file.\n",
    "\n",
    "## Important note\n",
    "\n",
    "Currently WebRTC can be used only by API keys the begin with:\n",
    "\n",
    "```\n",
    "sk-proj\n",
    "```\n",
    "\n",
    "and other keys may result internal server error  (500) on OpenAI server. For more details see:\n",
    "https://community.openai.com/t/realtime-api-create-sessions-results-in-500-internal-server-error/1060964/5\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "realtime_llm_config = autogen.LLMConfig.from_json(\n",
    "    path=\"OAI_CONFIG_LIST\",\n",
    "    temperature=0.8,\n",
    "    timeout=600,\n",
    ").where(tags=[\"gpt-4o-realtime\"])\n",
    "\n",
    "assert realtime_llm_config.config_list, (\n",
    "    \"No LLM found for the given model, please add the following lines to the OAI_CONFIG_LIST file:\"\n",
    "    \"\"\"\n",
    "    {\n",
    "        \"model\": \"gpt-4o-realtime-preview\",\n",
    "        \"api_key\": \"sk-prod*********************...*\",\n",
    "        \"tags\": [\"gpt-4o-realtime\", \"realtime\"]\n",
    "    }\"\"\"\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Before you start the server\n",
    "\n",
    "To run uvicorn server inside the notebook, you will need to use nest_asyncio. This is because Jupyter uses the asyncio event loop, and uvicorn uses its own event loop. nest_asyncio will allow uvicorn to run in Jupyter.\n",
    "\n",
    "Please install nest_asyncio by running the following cell."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!pip install nest_asyncio"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import nest_asyncio\n",
    "\n",
    "nest_asyncio.apply()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Implementing and Running a Basic App\n",
    "\n",
    "Let us set up and execute a FastAPI application that integrates real-time agent interactions."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Define basic FastAPI app\n",
    "\n",
    "1. **Define Port**: Sets the `PORT` variable to `5050`, which will be used for the server.\n",
    "2. **Initialize FastAPI App**: Creates a `FastAPI` instance named `app`, which serves as the main application.\n",
    "3. **Define Root Endpoint**: Adds a `GET` endpoint at the root URL (`/`). When accessed, it returns a JSON response with the message `\"WebRTC AG2 Server is running!\"`.\n",
    "\n",
    "This sets up a basic FastAPI server and provides a simple health-check endpoint to confirm that the server is operational."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from contextlib import asynccontextmanager\n",
    "\n",
    "PORT = 5050\n",
    "\n",
    "\n",
    "@asynccontextmanager\n",
    "async def lifespan(*args, **kwargs):\n",
    "    print(\"Application started. Please visit http://localhost:5050/start-chat to start voice chat.\")\n",
    "    yield\n",
    "\n",
    "\n",
    "app = FastAPI(lifespan=lifespan)\n",
    "\n",
    "\n",
    "@app.get(\"/\", response_class=JSONResponse)\n",
    "async def index_page():\n",
    "    return {\"message\": \"WebRTC AG2 Server is running!\"}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Prepare `start-chat` endpoint\n",
    "\n",
    "1. **Set the Working Directory**: Define `notebook_path` as the current working directory using `os.getcwd()`.\n",
    "2. **Mount Static Files**: Mount the `static` directory (inside `agentchat_realtime_webrtc`) to serve JavaScript, CSS, and other static assets under the `/static` path.\n",
    "3. **Set Up Templates**: Configure Jinja2 to render HTML templates from the `templates` directory within `agentchat_realtime_webrtc`.\n",
    "4. **Create the `/start-chat/` Endpoint**: Define a `GET` route that serves the `chat.html` template. Pass the client's `request` and the `port` variable to the template for rendering a dynamic page for the audio chat interface.\n",
    "\n",
    "This code sets up static file handling, template rendering, and a dedicated endpoint to deliver the chat interface.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "notebook_path = os.getcwd()\n",
    "\n",
    "app.mount(\"/static\", StaticFiles(directory=Path(notebook_path) / \"agentchat_realtime_webrtc\" / \"static\"), name=\"static\")\n",
    "\n",
    "# Templates for HTML responses\n",
    "\n",
    "templates = Jinja2Templates(directory=Path(notebook_path) / \"agentchat_realtime_webrtc\" / \"templates\")\n",
    "\n",
    "\n",
    "@app.get(\"/start-chat/\", response_class=HTMLResponse)\n",
    "async def start_chat(request: Request):\n",
    "    \"\"\"Endpoint to return the HTML page for audio chat.\"\"\"\n",
    "    port = PORT  # Extract the client's port\n",
    "    return templates.TemplateResponse(\"chat.html\", {\"request\": request, \"port\": port})"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Prepare endpoint for AG2 backend websocket\n",
    "\n",
    "1. **Set Up the WebSocket Endpoint**: Define the `/session` WebSocket route to handle audio streaming.\n",
    "2. **Accept WebSocket Connections**: Accept incoming WebSocket connections from clients.\n",
    "3. **Initialize Logger**: Retrieve a logger instance for logging purposes.\n",
    "4. **Set Up Realtime Agent**: Create a `RealtimeAgent` with the following:\n",
    "   - **Name**: `Weather Bot`.\n",
    "   - **System Message**: Introduces the AI assistant and its capabilities.\n",
    "   - **LLM Configuration**: Uses `realtime_llm_config` for language model settings.\n",
    "   - **Websocket**: Used by the RealtimeAgent backend to receive messages form WebRTC application.\n",
    "   - **Logger**: Logs activities for debugging and monitoring.\n",
    "6. **Register a Realtime Function**: Add a function `get_weather` to the agent, allowing it to respond with basic weather information based on the provided `location`.\n",
    "7. **Run the Agent**: Start the `realtime_agent` to handle interactions in real time.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "@app.websocket(\"/session\")\n",
    "async def handle_media_stream(websocket: WebSocket):\n",
    "    \"\"\"Handle WebSocket connections providing audio stream and OpenAI.\"\"\"\n",
    "    await websocket.accept()\n",
    "\n",
    "    logger = getLogger(\"uvicorn.error\")\n",
    "\n",
    "    realtime_agent = RealtimeAgent(\n",
    "        name=\"Weather Bot\",\n",
    "        system_message=\"Hello there! You are an AI voice assistant powered by Autogen and the OpenAI Realtime API. You can answer questions me about weather, jokes, or anything you can imagine. Start by saying 'How can I help you'?\",\n",
    "        llm_config=realtime_llm_config,\n",
    "        websocket=websocket,\n",
    "        logger=logger,\n",
    "        observers=[AudioObserver(logger=logger)],\n",
    "    )\n",
    "\n",
    "    @realtime_agent.register_realtime_function(name=\"get_weather\", description=\"Get the current weather\")\n",
    "    def get_weather(location: Annotated[str, \"city\"]) -> str:\n",
    "        logger.info(f\"Checking the weather: {location}\")\n",
    "        return \"The weather is cloudy.\" if location == \"Rome\" else \"The weather is sunny.\"\n",
    "\n",
    "    await realtime_agent.run()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Run the app using uvicorn"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "uvicorn.run(app, host=\"0.0.0.0\", port=PORT)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": []
  }
 ],
 "metadata": {
  "front_matter": {
   "description": "RealtimeAgent using websockets",
   "tags": [
    "realtime",
    "websockets"
   ]
  },
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.11"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
